I am guessing from the TOC that they are using pandas. My team quickly ran into memory spikes when multiple programs using pandas would run. We have since migrated to using ibis with a duckdb backend to smooth out the memory spikes.
These sort of things always seem to assume a fairly relaxed software environment.
In practice I’ve found the big corporates try hard to keep their excel files with financial data and their Python environments with pip & all those associated risks far apart. That’s if pip works at all & isn’t caught by a firewall
Most corporates, last I knew, didn't use Python outside of IT. Devs outside of IT would be using VBA.
However in Financial companies, Python and Excel have always been used together by devs and also quants.
And they tend to use Anaconda, and also like all their other package managers, they would host an in-house package repository and block the public one. That way only approved packages are used, and they only update packages as needed.
Many though have a policy of minimising Excel and rolling out formal platforms whether in-house or off the shelf, as Excel is regarded as a ongoing risk of in-accuracy as full editable at all times, lack of git/version control and so on.
>Many though have a policy of minimising Excel and rolling out formal platforms whether in-house or off the shelf, as Excel is regarded as a ongoing risk of in-accuracy as full editable at all times, lack of git/version control and so on.
Can confirm this practice in place at fortune 50 financial institutions. One in particular calls it "End-User Computing System Risk", meaning the end user created a business-critical "system" (i.e., a complex Excel file or Access database) on their own computer.
Depending on the size and the transformations you are doing, polars is worth checking out. Syntax is a bit different from pandas but the performance is really nice.
Looking at the content, if you are familiar at all with Python and basic programming, this provides very little new. I sometimes have to stuff massive Excel-abominations with 50k+ rows and rip data I need out of them with Pandas, but it only requires reading Pandas documentation (which is very good) a bit.
But perhaps this might be good if you know no programming and want to make your life easier.
Yesterday I asked an AI to generate a report as a CSV. But then I wanted it split into multiple sheets, and to add some formulas, so I asked it to create an XLSX, and it did (with Python). I'm imagining Microsoft embedding an AI agent and Python interpreter in their tools... no more need for a software dev, excel expert, or technical book author
Terrible title. Nothing to do with automating excel. From what I can tell it seems to be about ingesting spreadsheets into panda (and incredibly narrow use of Excel) and working outside of Excel.
What are best practices for using Excel as a front end for python tooling? I’ve got a use case where the business users are maintaining a complex spreadsheet and we need to hook some genuine optimization into it. It’s all fine and good if you assume the people will use the template perfectly, but hahahahahaha
There’s probably some ideal blend of locking regions, in-excel validation, in-python validation, and clean separation of human inputs and machine outputs. Has anyone figured out what that is already?
I am guessing from the TOC that they are using pandas. My team quickly ran into memory spikes when multiple programs using pandas would run. We have since migrated to using ibis with a duckdb backend to smooth out the memory spikes.
These sort of things always seem to assume a fairly relaxed software environment.
In practice I’ve found the big corporates try hard to keep their excel files with financial data and their Python environments with pip & all those associated risks far apart. That’s if pip works at all & isn’t caught by a firewall
Most corporates, last I knew, didn't use Python outside of IT. Devs outside of IT would be using VBA.
However in Financial companies, Python and Excel have always been used together by devs and also quants.
And they tend to use Anaconda, and also like all their other package managers, they would host an in-house package repository and block the public one. That way only approved packages are used, and they only update packages as needed.
Many though have a policy of minimising Excel and rolling out formal platforms whether in-house or off the shelf, as Excel is regarded as a ongoing risk of in-accuracy as full editable at all times, lack of git/version control and so on.
>Many though have a policy of minimising Excel and rolling out formal platforms whether in-house or off the shelf, as Excel is regarded as a ongoing risk of in-accuracy as full editable at all times, lack of git/version control and so on.
Can confirm this practice in place at fortune 50 financial institutions. One in particular calls it "End-User Computing System Risk", meaning the end user created a business-critical "system" (i.e., a complex Excel file or Access database) on their own computer.
This is just an advertisement for a book. And a terrible advertisement. No sample chapters to evaluate the book.
Yeah, a $50 book (physical) or $40 ebook
Yes it’s an add for a book but it does provide a sample chapter:
https://nostarch.com/download/samples/automate-excel-with-py...
Been working on a dashboard that takes in a bunch of different public data sources in different formats and pandas has truly been a godsend for this.
I've got csv, txt, xlsx in all different shapes and sizes and with just a few settings I can go through them quite easily and very fast as well.
Depending on the size and the transformations you are doing, polars is worth checking out. Syntax is a bit different from pandas but the performance is really nice.
My biggest bottleneck right now is upload speed but I'll keep it in mind thanks.
Looking at the content, if you are familiar at all with Python and basic programming, this provides very little new. I sometimes have to stuff massive Excel-abominations with 50k+ rows and rip data I need out of them with Pandas, but it only requires reading Pandas documentation (which is very good) a bit.
But perhaps this might be good if you know no programming and want to make your life easier.
in a world of ai agents, python is just an implementation detail that you don't need to know, between you and your data/business task
Yesterday I asked an AI to generate a report as a CSV. But then I wanted it split into multiple sheets, and to add some formulas, so I asked it to create an XLSX, and it did (with Python). I'm imagining Microsoft embedding an AI agent and Python interpreter in their tools... no more need for a software dev, excel expert, or technical book author
Terrible title. Nothing to do with automating excel. From what I can tell it seems to be about ingesting spreadsheets into panda (and incredibly narrow use of Excel) and working outside of Excel.
its just a really badly written ad
"Stop wasting your time on Excel and waste even more coding an Excel parser in Python!"
I prefer DuckDB
How do you automate Excel with a RDBMS?
DuckDB isn't an rdbms, bit the answer is the same: ODBC or OLAP. There's a "data connection wizard".
What are best practices for using Excel as a front end for python tooling? I’ve got a use case where the business users are maintaining a complex spreadsheet and we need to hook some genuine optimization into it. It’s all fine and good if you assume the people will use the template perfectly, but hahahahahaha
There’s probably some ideal blend of locking regions, in-excel validation, in-python validation, and clean separation of human inputs and machine outputs. Has anyone figured out what that is already?