MS Access bug/fix - importing large CSV text files
Bad? Brilliant?
You can rate this post.
Register or login now and
tell us what you think.
URGENT for users who import large data files into MS ACCESS 2007 data base.

Microsoft admits a problem, "Access 2007 takes more time to import large [30 megabyte] .csv files than Access 2003 does," but claims slow import is fixed in MS Office 2007 Service Pack 1 (SP1).

Do you have/need MS Office Service Pack 1 (SP1)?

In MS Access 2007 it is not just "Help/About" anymore....

If you are in MS Access 2007:

1. Click the Microsoft Office Button (round icon upper left corner), and then click Access Options (lower right corner).

2. Click Resources (last item of list), and then click About (last item of list).

3. Check for Updates button (upper right) or Get Updates (top of list)



The slow CSV import issue is one of many listed in Microsoft Knowledge Base article kb 942378.
http://support.microsoft.com/kb/942378

QUOTE from Microsoft kb 942378

"The 2007 Microsoft Office suites SP1 [Service Pack 1]
* * *
also fixes the following issues that were not previously documented in a Microsoft Knowledge Base article:
* * *
Access 2007 takes more time to import large .csv files than Access 2003 does

When you try to import a large .CSV file, such as a file that is larger than 30 megabytes (MB),
Access 2007 appears to stop responding at the import dialog box.

After several moments, you may receive the following error message:

File sharing lock count exceeded. Increase MaxLocksPerFile registry Entry

When you click OK to close the error message, you can usually finish the import process. All records are imported successfully."



http://support.microsoft.com/kb/942378


====
The slow CSV/TSV import issue has been encountered by the Orange County DEC IT Committee.

Documented by Jim Callahan
on behalf of
Orange County DEC Information Technology (IT) Committee, Doug De Clue, Chair

Reader Comments
  
lol
By Mikey Jul 23rd 2008 at 11:08 am EDT
Seems odd in this forum. I haven't used Access in years, after that bad SQL worm. Me and my company use linux, and build our own databases.
Re: lol
By Jim Callahan Jul 23rd 2008 at 11:25 am EDT
Most campaigns are relying on pre-built voter files such as VAN or VoteBuilder.

MS Access is handy on Windows (not available natively on Apple OS/X or Linux).

OpenOffice.org Base doesn't have this problem yet, because it doesn't even have a data import module, they sugguest you import to OpenOffice.org Calc and cut and paste data -- not exactly feasible for 30 megabyte files!

MySQL has problems importing some tab delimited files from Florida Division of Elections FVRS (Florida Voter Registration System).

Compared to desktop data bases such as MS Access, server data bases are unbundled mix and match affairs. Query Builders, Report Writers, Forms and even data import are all add on modules.

Open source data import add-ons for open source server data bases such as MySQL or PostgreSQL include Talend and Pentaho/Kettle. I have not tried them, yet, in this context.

Once data is in data base it can be moved to another data base using ADO, ODBC or JDBC connectors.

Jim Callahan
Orlando, FL
Re: lol
By Mikey Jul 23rd 2008 at 11:36 am EDT
I'm guessing you and I have similar jobs. I build data analysis models for many companies. I have never dealt with voter data though. When I used access I never went above 10 megs of data. By that point I wrote my own perl scripts to import and parse files.
PERL
By Jim Callahan Jul 23rd 2008 at 1:41 pm EDT
Perl is a good choice (Practical Extraction and Reporting Language) if you are near a university or other organization that uses it. It is open source (free download), fast, but somewhat cryptic to newbees. Python is easier to read.

Do you know where to find good sample scripts? CPAN.org (Comprehensive Perl Archive Network) perhaps?

In a political campaign, you can't count on anyone knowing a particular programming language (or any programming language) and you generally don't have time to teach much beyond task specific information.

In politics, outside of specialized consulting, everyone, except John McCain, knows how to send email. Almost everyone can type something in MS Word. Most young people can use MS Excel. A few can use MS Access. A tiny few are professional programmers, engineers or statisticians who are involved in politics and can program and use stuff beyond desktop apps.

Jim Callahan
Orlando, FL
Re: PERL
By Jim Callahan Jul 23rd 2008 at 2:19 pm EDT
Here is a link to a sample PERL program for importing data to MySQL:

Link

Jim Callahan
Orlando, FL