Meeting with Bryan. I showed him some of the changes to the online scheduling stuff. He helped build the original functionality for reoccurring and special events. We then switched gears and did some planning on dealing with huge record sets and bigger reports and big data exports. Basically, we have the data in the database, but it takes too long to show the whole thing (all of the records) in the browser. Imagine asking for 20,000 records at once. The database servers that up like a snack. However, the browsers struggle because that is a big bite to chew and virtually it is like putting an elephant through a straw. It's painful.

We did lots of drawing, looking at code, samples, existing pages that do mini versions of this, etc. Good planning session. Bryan has already been playing with some code and doing some research. That is awesome. We will continue to keep playing until we feel like we have a good plan and a good mini version of what we are wanting before trying to tie it into live reports.

The discussion turned towards two major options. One is AJAX or asynchronous downloading of the data. We have to respond quickly and then show progress while pulling in the other (thousands) records. The other major option is to offload it, meaning the search and building the export, by using a queue of sorts. Basically, we push the search criteria to the queue, get a total record count, and then start building that behind the scenes. If the user wants to check, we can show progress of how many records have been added as compared with the total record count. We then notify the user when it is done. That way they don't have to wait, and they can be doing something else instead of just waiting and hoping that it will not timeout.

Bryan and I made a plan for doing some local testing on a report with 70,000 ish records and then moving it over to live once we've knocked it around a bit. We went over a few other details, but Bryan is going to start playing with some scratch files. Making progress.