Recent Entries 10
- pattern tip 27d agoargparse CLI with per-item error isolation and monkeypatch testing patternWhen building a CLI that processes a list of items (tickers, files, URLs), one bad item can crash the whole run. Also, testing argparse CLIs with pytest requires patching module-level names and capturing stdout.
- pattern minor 112d agoA way to reference other DB without hardcoding its nameMy customer uses pair of databases (SQL Server) where one references the other. Let's say `DataDb` and `ConfigDb`. `ConfigDb` contains hundreds of SPs that references `DataDb` like `[DataDb].[dbo].[the_object]`. This pair of databases is deployed to their customers together with some applications. For development and testing I need to have locally pairs for several customers at once and run the applications against the right pair just by changing the connection string. When I restore the customer's databases I use suffix of the customer, like `DataDb_Cust1`, `ConfigDb_Cust1`. The problem is that stored procs from ConfigDb have hardcoded the name of the referenced database (there's always `[DataDb].[dbo].[the_object]` which I need to change to `[DataDb_Cust1].[dbo].[the_object]`). I could go through all the object (SPs, functions, views) and rename the referenced DB name. I already have some scripts to automate that. But is there a better way (e.g. just use some alias defined on SQL server level or something like that) so that the referenced DB name does not have to be hardcoded in SPs? I looked at synonyms, but it turns out they are not supported by Microsoft Entity Framework - ORM used by the application. I'd need a solution that would work with EF6. However I can verify it by myself. Any other solution? The size of DB varies, but typically the DataDb is between 1 and 8GB, the ConfigDb is quite small. No Enterprise features, but it uses SQLCLR assemblies, for some customers even linked servers to access other DBs in distributed transactions (using MSDTC) but it is rare case. I have now 6 sets and expect at most 20 in near future. The DataDb uses `FILESTREAM`, which is not supported by LocalDb.
- pattern minor 112d agoTool to test RDBMS SQL dialect for ANSI SQL standard complianceWe are developing an interface to our internal data storage and manipulation tool that would allow people to use SQL dialect. We decided that it would be perfect to make our SQL dialect as close to the standard as possible. Currently I'm trying to find a tool that would test our implementation of query language and make a verdict about its compliance and list missing details. Have anyone ever encountered anything like this?
- pattern minor 112d agoDatabase staging environment for non-IT personnelI'm in the process of proposing a database staging environment to my IT department. The idea is that a non-IT person like me (public works data analyst) would have a place to test solutions, and then either implement them in the live environment myself, or ask IT to implement them if needed. There are a few reasons/scenarios where this environment would be beneficial: - I have some basic database privileges in our live database environment (`create table`, `create view`, etc.). I make schema changes about once a week, but it seems insane for me to test and implement these changes in a live environment. There are countless dependencies on the database, so if something goes wrong, it could be disastrous. I'd much rather test things out ahead of time in a separate environment. - I don't have some of the more advanced privileges like `create trigger` or `create function` in the live database. This is fine, but I do have a few problems that could be solved by triggers and/or functions. I plan on proposing that I be granted these permissions in the staging environment so that I can develop and test some ideas, and if they work, propose that IT implement them in the live environment. - In general, my IT department doesn't have the time or resources to develop solutions for me. It's really that simple. So if I can do the legwork myself, then my problems are much more likely to be solved. The 'staging environment for non-IT personnel' seems like a sound enough approach to me, but to be honest, I just made the idea up. I have no idea how this is typically done in the IT/database world. Is there any sort of established IT/Database practice that would fit this scenario? (Am I on the right track when proposing a database staging environment for non-IT personnel?)
- debug minor 112d agoDROP DATABASE statement cannot be used inside a user transactionNot really sure if this question belongs here, but I hope someone could help me out. I've made integration tests going all the way down to the database (using mssql localDB). I want each test to run independently with it's own data - I want to reseed the database with my fake data before each test is running. I tried to implement it with transactions without success. Here is how I tried to pull it off: ``` public class TestDbInitializer : DropCreateAlways() { public static List Items; public override Seed(DbContext context) { Items = new List(); // Adding items // .. Items.ForEach(x => context.Add(x)); context.SaveChanges(); } } public class BaseTransactionsTests { private TransactionScope _scope [TestInitialize] public void Initialize() { _scope = new TransactionScope(); } [TestCleanup] public void Cleanup() { _scope.Dispose(); } } [TestClass] public class IntegrationTests : BaseTransactionsTests private IDependenciesContainer _container; public static void AssemblyInit(TestContext context) { Database.SetInitializer(new TestDbInitializer()); _container = new DependenciesContainer(); // Registers all my application's dependencies _container.RegisterAll(); } [TestInitialize] public void Initialize() { using (var context = new MyContext("TestsDatabase")) { context.Initialize(true); } } [TestMethod] public void TestAddItem() { var controller = _container.Resolve(); var result = controller.AddItem(new Item({Name = "Test"})) var goodResult = result as OkNegotiatedResult(); if (result == null) Assert.Fail("Bad result") using (var context = new MyContext("TestsDatabase")) { Assert.AreEqual(context.Items.Count, TestDbInitializer.Items.Count + 1) } } ``` I use my dependency injector in my tests, registering all dependencies once (AssemblyInitialize). I created a DB in
- pattern moderate 112d agoCan I restore all tables but not the stored procedures?I have a testing database that I restore regularly from a production backup file in SQL Server 2008 R2 to get new data. We often edit\change or update the stored procedures in the testing DB to test the data compared to the production's stored procedures! Is there away to only refresh\restore the table(s) and not the stored procedures? so I don't have to re-edit them back to the changes that I need? Both of these databases (Production and Testing) are in one server. My DB size is about 5 GB and we can't afford to have third party tools.
- pattern moderate 112d agoIs it ever ok to not test a feature?Is there any point at which you become so familiar with your language/database/system that there is no need to test a new feature/configuration/query/etc. by contained/simulated testing before implementing it in your system (especially concerning a feature that modifies data)? Or is it always essential to test a new query by simulation in a test environment? To specify further, it is clear that it is always safest to test. However, is there a way to determine when the risk is so minimal that testing is not worth the effort? Another way of phrasing that: when or is it ever professional practice to take a measured risk to implement a feature? Also, let's assume that everything is backed-up, so, worst-case scenario, the data could with some effort be restored. Can someone cite specific, expert experience to address this? Please include references where appropriate/possible.
- pattern minor 112d agoCreating a 'shadow' database in MySQLA while back, a friend told me of a facility that would allow us to "shadow" a production database using a local database. The term shadow may be incorrect (so please correct me if so) What I understand, and want to do is: - When I make an update / insert, it only impacts the database that is local. - I make a select statement, the return is first the data from the database that is shadowing, then the production data is filled in. - table modifications only impact the database that is shadowing - clear out (roll back transactions?) to get the local to a neutral point. - would prefer not to have to copy the database over, but pull from the live data if possible. Since this is not designed for production, performance is not really an issue. The use case in mind is that I would have said database for local / testing, and a quick way to revert all the changes that I had done, if they need to be.
- pattern minor 112d agoA little clarification on how 'LIMIT' worksNot being a full blooded member of the DBA clan, but more of a dev clan, I find myself trying to figure out some slow running queries. The subject of my question however is not 'Why my queries are slow' , but one of wanting to know how the 'LIMIT' command in MySql works. At the moment, I'm still trying to troubleshoot the server performance itself, so as a way of giving it a little SQL to chew on (Just to make it do some work) I'm using the following: ``` SELECT * FROM LIMIT 10 ``` The table I'm using is a live table with an excess of 50Mil rows in it, and as I'm only conducting tests at the moment my expectation is this will return the first 10 records off the top of the table, and would easily do so in under 10 minutes. However... I'm seeing this SQL still running after 30 mins or more, and appearing not to do anything. What I want to clarify specifically is this: Does MySql attempt to fetch the entire table from a DB before then just returning the first 10 rows OR Does MySql fetch rows as it receives them and stop after it counts that it's recieved 10 rows Cheers Shawty
- pattern moderate 112d agoCan PostgreSQL support integration test with some kind of throwaway overlay?It's a common problem to write integration tests that include a database. If the test changes the database then it could effect other tests or the next run of itself. I know that I could wrap my test in a transaction and rollback the transaction after the test run. But it would be very nice if PostgreSQL could provide some kind of global snapshoting or throwaway overlay. In an ideal case such a feature would cover all state of the database including schemas and stored procedures.