Re-code the units a run failed on

A coding run over a real corpus rarely comes back complete. Requests time out or are rate-limited, a provider refuses a text on one pass and codes it on the next, an endpoint accepts a schema and ignores it for a few units. The failed units sit in the qlm_coded object as NA rows, listed by qlm_failures(). qlm_backfill() re-codes only those units and merges what comes back into the original object. Everything that succeeded the first time is left exactly as it was.

Usage

qlm_backfill(x, ..., model = NULL, passes = 2L)

Arguments

x: qlm_coded; a coded object produced by qlm_code() or qlm_replicate().
...: optional overrides passed to qlm_code() for the backfill passes, such as params, max_active or on_error. Any setting not overridden is restored from the original run, as qlm_replicate() does, with the same rule for credentials and endpoint settings when the provider changes. The codebook, batch, name and backfill cannot be set: passes is the only bound on the number of passes. Nor can include_tokens and include_cost: usage is recorded as the run recorded it, so that the merged columns mean one thing. prices may be given, to cost the passes where the run's own rates do not carry (a batch-coded run's rates do not cover its parallel passes), and needs the run to have recorded token counts and cost of its own to add to.
model: character or NULL; the model for the passes, in the form used by qlm_code(). NULL (default) uses the run's own model.
passes: A single positive integer giving the maximum number of backfill passes. Default is 2. This counts total passes, not additional retries. Backfilling stops early when a pass recovers no units, since the failures that remain are then evidently not transient.

Value

x, with the recovered units filled in and backfill added to its object metadata. The run name, parent, codebook and inputs are unchanged.

Details

By default the passes use the run's own model, codebook and settings, so the result is what the run should have produced. A different model can be given, for units the original model consistently refuses or cannot fit in its context window; the object then records which units were coded by which model, and print() and qlm_trail() say so, since a result coded by two instruments has to be disclosed as one.

Which units are re-coded is decided afresh on every pass, from the object as it then stands, by the same test qlm_failures() uses: a unit that carries an .error, or whose required scalar properties are all NA. Two kinds of failure are left alone, because re-sending the same request cannot change the outcome:

a text the provider rejected as longer than the model's context window, unless the model or endpoint changes;
a response cut off at the max_tokens limit, unless the model or endpoint changes or the backfill raises the limit, by passing params(max_tokens = ) higher than the run's own. Other params leave the limit where it was, and so leave those units alone.

Content refusals are deliberately retried. They look deterministic and are not: the same document is refused on one pass and coded on the next, at more than one provider.

Each pass is an ordinary qlm_code() call over the failed units, on the path the original run took (a run that fell back to JSON mode is backfilled in JSON mode; with a different endpoint the path is chosen afresh), and always as a parallel call: a run coded through the batch API is backfilled through the parallel API, with the same model and settings, and any batch-only arguments (path, wait, ignore_hash) set aside. A pass that fails outright on the first attempt is an error, since nothing has been gained yet and the cause is most likely configuration; on a later pass it is a warning, and what earlier passes recovered is kept. The failed pass is still recorded, with the units it attempted, no recoveries and the error, since the provider may have billed it, and for the same reason the token and cost columns of the units it attempted become NA: how much was billed is not known, so no total for them is.

Units are identified by .id throughout: the failed units' inputs are looked up by .id, so an object whose rows have been reordered or subset is backfilled correctly, and the merge is by .id. Rows keep their order; a unit is replaced only when the retry produced a usable coding, so a retry that failed again never overwrites anything, though its .error is recorded as the latest reason. Token and cost columns, when present, are summed across all attempts, since a failed request may still have been billed; a total is NA when any attempt's figure is, because NA means the provider did not report it, not that nothing was billed, so a retry's known figure cannot stand in for the whole. The passes are recorded in the object metadata as backfill, one entry per pass with its timestamp, the model if it or the endpoint differed from the run's, the overrides, the .ids attempted and recovered, where its cost came from when that was not where the run's did, and for a pass that failed outright its error, so the result can say which of its rows came from which pass and which model. A pass whose cost came from somewhere else than the run's, other supplied rates, ellmer's own table where the run rested on supplied rates, or nowhere where the run was priced, is disclosed by print() and qlm_trail() beside the run's own cost note, since part of the cost column then rests on it. qlm_trail() redacts any credential among a pass's overrides as it does the run's own, and a pass replayed from a trail does not send a redacted value. qlm_replicate() replays these passes on a replication, so that a replication of a completed run is completed on the same terms.

Examples

# A run that came back incomplete, and what qlm_backfill() made of it. Both
# were coded once and saved with the package (see data_creation/ in the
# source), so they can be looked at without a key.
examples <- readRDS(system.file("extdata", "example_objects.rds", package = "quallmer"))
incomplete <- examples$example_coded_incomplete
incomplete
#> # quallmer coded object
#> # Run:      example_incomplete
#> # Codebook: Sentiment with evidence
#> # Model:    anthropic/claude-haiku-4-5
#> # Units:    8 (4 scored, 4 failed)
#> # Notes:    Coded with a deliberately short request timeout and max_tokens = 90, so that the run came back incomplete
#> 
#> # A tibble: 8 × 5
#>   .id          sentiment rating evidence                              .error    
#> * <chr>        <fct>      <int> <chr>                                 <list>    
#> 1 3275_2.txt   neg            1 "Lets just say that from now on, we … <NULL>    
#> 2 3150_1.txt   NA            NA  NA                                   <qllmr_t_>
#> 3 3918_1.txt   NA            NA  NA                                   <qllmr_t_>
#> 4 7530_1.txt   neg            1 "\"to call this the next cult classi… <NULL>    
#> 5 4247_8.txt   pos            7 "One very strange piece of cinema. Y… <NULL>    
#> 6 7227_7.txt   NA            NA  NA                                   <httr2_fl>
#> 7 8877_8.txt   pos            9 "The one thing I really can't seem t… <NULL>    
#> 8 11413_10.txt NA            NA  NA                                   <qllmr_t_>
qlm_failures(incomplete)
#> # A tibble: 4 × 3
#>   .id          reason                                                 .error    
#>   <chr>        <chr>                                                  <list>    
#> 1 3150_1.txt   "The response used the whole max_tokens limit of 90 a… <qllmr_t_>
#> 2 3918_1.txt   "The response used the whole max_tokens limit of 90 a… <qllmr_t_>
#> 3 7227_7.txt   "Failed to perform HTTP request.\nCaused by error:\n!… <httr2_fl>
#> 4 11413_10.txt "The response used the whole max_tokens limit of 90 a… <qllmr_t_>

# What qlm_backfill(incomplete) returned: the timed-out unit re-coded, the
# responses cut off at max_tokens left alone, and the pass on record
filled <- examples$example_coded_backfilled
filled
#> # quallmer coded object
#> # Run:      example_incomplete
#> # Codebook: Sentiment with evidence
#> # Model:    anthropic/claude-haiku-4-5
#> # Units:    8 (5 scored, 3 failed)
#> # Backfill: 1 pass, recovered 1 of 1
#> # Notes:    Coded with a deliberately short request timeout and max_tokens = 90, so that the run came back incomplete
#> 
#> # A tibble: 8 × 5
#>   .id          sentiment rating evidence                              .error    
#>   <chr>        <fct>      <int> <chr>                                 <list>    
#> 1 3275_2.txt   neg            1 "Lets just say that from now on, we … <NULL>    
#> 2 3150_1.txt   NA            NA  NA                                   <qllmr_t_>
#> 3 3918_1.txt   NA            NA  NA                                   <qllmr_t_>
#> 4 7530_1.txt   neg            1 "\"to call this the next cult classi… <NULL>    
#> 5 4247_8.txt   pos            7 "One very strange piece of cinema. Y… <NULL>    
#> 6 7227_7.txt   pos            8 "ah man this movie was funny as hell… <NULL>    
#> 7 8877_8.txt   pos            9 "The one thing I really can't seem t… <NULL>    
#> 8 11413_10.txt NA            NA  NA                                   <qllmr_t_>
qlm_failures(filled)
#> # A tibble: 3 × 3
#>   .id          reason                                                 .error    
#>   <chr>        <chr>                                                  <list>    
#> 1 3150_1.txt   The response used the whole max_tokens limit of 90 an… <qllmr_t_>
#> 2 3918_1.txt   The response used the whole max_tokens limit of 90 an… <qllmr_t_>
#> 3 11413_10.txt The response used the whole max_tokens limit of 90 an… <qllmr_t_>
qlm_meta(filled, "backfill", type = "object")
#> [[1]]
#> [[1]]$timestamp
#> [1] "2026-09-04 13:01:38 UTC"
#> 
#> [[1]]$model
#> NULL
#> 
#> [[1]]$overrides
#> list()
#> 
#> [[1]]$attempted
#> [1] "7227_7.txt"
#> 
#> [[1]]$recovered
#> [1] "7227_7.txt"
#> 
#> 

if (FALSE) { # \dontrun{
filled <- qlm_backfill(incomplete)

# Responses cut off at the output limit are retried only with a higher one
filled <- qlm_backfill(filled, params = ellmer::params(max_tokens = 2000))

# Units one model refuses or cannot fit, coded by another; the result
# records which units came from which model
filled <- qlm_backfill(filled, model = "deepseek/deepseek-chat")
} # }

Usage

Arguments

Value

Details

See also

Examples