Alternative to Excel for handling large datasets

[liiamra]

Hello Everyone,

I have few datasets containing millions of records.

Is there any program that can be used for the purpose of handling and managing such large datasets? Kindly note that each dataset is contained in one table.

ATM, I am using Excel, but the problem is that I have to split the dataset first, and then always write codes, even when a very simple figure is needed. I tried OpenRefine for data analysis. It is a very good program, but it is not very efficient in handling large datasets.

I don't really have problem with storing the data in one table; SQLite or others can do it, but I need a tool that allows me to analyze and handle the data.

I tried Access, but it failed with my datasets.

Any advice is highly appreciated,
Thanks all,
LA

 

[SydneyGeek]

If you like and are comfortable with Access for building the interface, stick with it.
As xenou suggested, hook it up to a 'proper' back-end database to handle the volume. Really doesn't matter which database you hook it to. I have used front ends with SQL Server or Oracle back ends.

Denis

 

[Cindy Ellis]

If you only have 52 columns, could you split the data not between sheets, but into multiple sets of columns on the same sheet in Excel? If you're developing a UI to handle the data anyway, you might also be able to manage the data more easily on the same sheet rather than split between sheets.
Regarding Minitab, I don't think there's a mechanism to develop a UI.
Best of luck with your project.

 

[RoryA]

I wasn't suggesting Excel for the data storage - you would require a database of some sort for that, but PowerPivot can handle millions of records so you can do the analysis and manipulation in Excel/PP.

 

[liiamra]

Dear All,

Really sorry, but I just noticed your replies after I logged-in again, though I thought I would receive an email.

Anyways, Thanks again for your suggestions.

Denis, Cindy, Rory, and everyone else. Thanks alot for your comments and hints; they are indeed very helpful.

Regards+
liiamra