patternsqlMinor
Using desktop computers to manage an SQLite database on a fileserver?
Viewed 0 times
fileserverdesktopsqlitemanagedatabasecomputersusing
Problem
Sometimes, it can take months to get a proper database server up and running in large companies. They have all kinds of policies, which means that this process takes months. An alternative in the meantime is to place an SQLite database directly on a file server. This works nicely for a limited number of users (I have had success with serving 100 users, using one such database), and it is only recommended for non-critical data. Now I have been thinking about ways to get a database, which is placed on a file server to be able to handle, say ten times as many users. The basic idea I have is to utilize the many, mostly idle desktop computers around the company. Again, the data would be non-critical, database access could be slow, and uptime of 99.0% is sufficient.
Do you see any way of utilizing these, mostly ideal desktop computers to help reduce the load on the file server? Do you have any idea of DBMS, which would be most suited, SQLite was just my first thought. It would be best with an open-source, one so I could hire someone to adapt it for this purpose. Alternatively, I would be interested in building a DBMS for this purpose from scratch.
I know that any experienced DBA will pull their hair out over this question, but would really have value due to the policies of these companies.
Do you see any way of utilizing these, mostly ideal desktop computers to help reduce the load on the file server? Do you have any idea of DBMS, which would be most suited, SQLite was just my first thought. It would be best with an open-source, one so I could hire someone to adapt it for this purpose. Alternatively, I would be interested in building a DBMS for this purpose from scratch.
I know that any experienced DBA will pull their hair out over this question, but would really have value due to the policies of these companies.
Solution
The issue with this is that as you scale SQLite, you run into the fundamental problem that it doesn't have row-level locks, only table locks. The "thing" you run out of first isn't a raw resource like CPU or IO or memory - it's that your app will get bogged down in lock management. You can have lots and lots of users sharing an SQLite database, but only your readers can tolerate waiting for your writers. If this database is going to be on a a shared drive, then that will only make your scalability problems worse as it introduces additional latency, making the lock acquire/release cycle longer!
Also, in a business, there is no such thing as "non-critical data" unless you are literally talking about people MP3 or recipe collections! If people use it to get their work done and for whatever reason it "goes away" then they will wonder why, and perhaps escalate it up their reporting line. I totally sympathize with how long it takes to get a "proper" database server (be aware that the DBA feels exactly the same way, when he's waiting for the procurement department to buy the servers!) but unless you are prepared to support something forever, don't deploy it. Or if you do, develop it with something like the free versions of Oracle or SQL Server, so that when the time comes, it can easily be migrated into the datacentre.
Also, in a business, there is no such thing as "non-critical data" unless you are literally talking about people MP3 or recipe collections! If people use it to get their work done and for whatever reason it "goes away" then they will wonder why, and perhaps escalate it up their reporting line. I totally sympathize with how long it takes to get a "proper" database server (be aware that the DBA feels exactly the same way, when he's waiting for the procurement department to buy the servers!) but unless you are prepared to support something forever, don't deploy it. Or if you do, develop it with something like the free versions of Oracle or SQL Server, so that when the time comes, it can easily be migrated into the datacentre.
Context
StackExchange Database Administrators Q#3045, answer score: 8
Revisions (0)
No revisions yet.