Programming Essentials Python - Batch Operations - Best Practices Batch Loading

Let us go through some of the best practices to perform batch load.

  • We should minimize the number of connections to the database.
  • We should avoid executing queries using hard coded values. Rather, we should prefer bind variables.
  • Too much committing is bad as commit incurs overhead.
  • If we have to load a considerable amount of data, we should consider committing every 1,000 records or 10,000 records or even more based upon the capacity of the database.
  • Most of the mainstream databases perform direct path I/O or batch load which might perform better compared to looping, inserting, and committing data. In some cases, we can use those features.

Watch the video tutorial here