ℹ️ Select 'Choose Exercise', or randomize 'Next Random Exercise' in selected language.

Choose Exercise:
Timer 00:00
WPM --
Score --
Acc --
Correct chars --

PL/pgSQL Cursor Batch Processing

PL/pgSQL

Goal -- WPM

Ready
Exercise Algorithm Area
1CREATE OR REPLACE FUNCTION process_records_in_batches(
2p_batch_size INTEGER
3) RETURNS VOID AS $$
4DECLARE
5-- Declare a cursor to select records
6cur_records CURSOR FOR
7SELECT id, name FROM source_table ORDER BY id;
8
9v_record_id INTEGER;
10v_record_name VARCHAR(100);
11v_batch_count INTEGER := 0;
12v_total_processed INTEGER := 0;
13
14BEGIN
15-- Open the cursor
16OPEN cur_records;
17
18LOOP
19-- Fetch a batch of records
20FETCH FOR cur_records INTO v_record_id, v_record_name;
21
22-- Check if the cursor is empty
23EXIT WHEN NOT FOUND;
24
25-- Process the fetched record
26RAISE NOTICE 'Processing record: ID=% Name=%', v_record_id, v_record_name;
27-- In a real scenario, perform some operation on the record
28v_batch_count := v_batch_count + 1;
29v_total_processed := v_total_processed + 1;
30
31-- If batch size is reached or it's the last record fetched
32-- (and we've fetched at least one record in this iteration)
33IF v_batch_count = p_batch_size THEN
34RAISE NOTICE 'Batch of % processed. Total processed: %', p_batch_size, v_total_processed;
35-- Perform commit or other batch-level operations here
36v_batch_count := 0; -- Reset batch counter
37END IF;
38
39-- If this is the last record fetched and it wasn't a full batch,
40-- process the partial batch.
41-- This condition is implicitly handled by the loop structure and the
42-- final check after the loop if needed, but explicit check is clearer.
43-- The EXIT WHEN NOT FOUND handles the end of cursor.
44END LOOP;
45
46-- Handle any remaining records in the last partial batch
47IF v_batch_count > 0 THEN
48RAISE NOTICE 'Final partial batch of % processed. Total processed: %', v_batch_count, v_total_processed;
49-- Perform commit or other batch-level operations for the final batch
50END IF;
51
52RAISE NOTICE 'All records processed. Total: %', v_total_processed;
53
54-- Close the cursor
55CLOSE cur_records;
56END;
57$$ LANGUAGE plpgsql;
Algorithm description viewbox

PL/pgSQL Cursor Batch Processing

Algorithm description:

This PL/pgSQL function demonstrates efficient processing of large datasets by fetching and processing records in manageable batches. It utilizes a cursor to iterate through rows from a source table and accumulates them until a predefined batch size is met. This approach is vital for preventing memory exhaustion and improving performance when dealing with millions of records, as it allows for periodic commits or other batch-specific operations.

Algorithm explanation:

The function `process_records_in_batches` opens a cursor to iterate over `source_table`. It fetches records one by one within a `LOOP`. A counter `v_batch_count` tracks the number of records in the current batch. When `v_batch_count` reaches `p_batch_size`, a message is logged, and the counter is reset. The loop terminates when `FETCH` returns `NOT FOUND`. After the loop, if `v_batch_count` is greater than zero, it signifies a final partial batch that is also processed. This ensures all records are accounted for. The time complexity is O(N), where N is the total number of records, as each record is fetched and processed once. The space complexity is O(1) as it only uses a few variables to manage the batching and counters.

Pseudocode:

FUNCTION process_records_in_batches(batch_size):
  DECLARE cursor FOR SELECT ... FROM source_table ORDER BY ...
  batch_count = 0
  total_processed = 0

  OPEN cursor
  LOOP:
    FETCH cursor INTO record_vars
    IF NOT FOUND:
      EXIT LOOP

    PROCESS record_vars
    increment batch_count
    increment total_processed

    IF batch_count == batch_size:
      LOG 'Batch processed'
      // Perform batch-level operations (e.g., COMMIT)
      reset batch_count
  END LOOP

  IF batch_count > 0:
    LOG 'Final partial batch processed'
    // Perform batch-level operations for final batch

  LOG 'All records processed'
  CLOSE cursor
END FUNCTION