data efficiency and context-aware modeling math details

content/04.interpretations.md

@@ -0,0 +1,43 @@
+## Context-Adaptive Interpretations of Context-Invariant Models
+
+In the previous section, we discussed the importance of context in model parameters. 
+Such context-adaptive models can be learned by explicitly modeling the impact of contextual variables on model parameters, or learned implicitly in a model containing interaction effects between the context and the input features.
+In this section, we will focus on recent progress in understanding how context influences interpretations of statistical models, even when the model was not originally designed to incorporate context.
+
+TODO: Discussing the implications of context-adaptive interpretations for traditional models. Related work including LIME/DeepLift/DeepSHAP.
+
+### Context-Aware Efficiency Principles and Design
+
+The efficiency of context-adaptive methods hinges on several key design principles that balance computational tractability with statistical accuracy. These principles guide the development of methods that can scale to large datasets while maintaining interpretability and robustness.
+
+Context-aware efficiency often relies on sparsity assumptions that limit the number of context-dependent parameters. This can be achieved through group sparsity, which encourages entire groups of context-dependent parameters to be zero simultaneously [@Yuan2006ModelSA], hierarchical regularization that applies different regularization strengths to different levels of context specificity [@tibshirani1996regression;@Gelman2006DataAU], and adaptive thresholding that dynamically adjusts sparsity levels based on context complexity.
+
+Efficient context-adaptive inference can be achieved through computational strategies that allocate resources based on context. Early stopping terminates optimization early for contexts where convergence is rapid [@Bottou2016OptimizationMF], while context-dependent sampling uses different sampling strategies for different contexts [@Balseiro2018ContextualBW]. Caching and warm-starting leverage solutions from similar contexts to accelerate optimization, particularly effective when contexts exhibit smooth variation [@Boyd2011DistributedOA].
+
+The design of context-aware methods often involves balancing computational efficiency with interpretability. Linear context functions are more interpretable but may require more parameters, while explicit context encoding improves interpretability but may increase computational cost. Local context modeling provides better interpretability but may be less efficient for large-scale applications. These trade-offs must be carefully considered based on the specific requirements of the application domain, as demonstrated in recent work on adaptive optimization methods [@Kingma2014AdamAM].
+
+### Adaptivity is bounded by data efficiency
+
+Recent work underscores a practical limit: stronger adaptivity demands more informative data per context. When contexts are fine-grained or rapidly shifting, the effective sample size within each context shrinks, and models risk overfitting local noise rather than learning stable, transferable structure. Empirically, few-shot behaviors in foundation models improve with scale yet remain sensitive to prompt composition and example distribution, indicating that data efficiency constraints persist even when capacity is abundant [@Brown2020LanguageMA; @Wei2022EmergentAO; @Min2022RethinkingTR]. Complementary scaling studies quantify how performance depends on data, model size, and compute, implying that adaptive behaviors are ultimately limited by sample budgets per context and compute allocation [@Kaplan2020ScalingLF; @Hoffmann2022TrainingCO; @Arora2024BayesianSL]. In classical and modern pipelines alike, improving data efficiency hinges on pooling information across related contexts (via smoothness, structural coupling, or amortized inference) while enforcing capacity control and early stopping to avoid brittle, context-specific artifacts [@Bottou2016OptimizationMF]. These considerations motivate interpretation methods that report not only attributions but also context-conditional uncertainty and stability, clarifying when adaptive behavior is supported by evidence versus when it reflects data scarcity.
+
+#### Formalization: data-efficiency constraints on adaptivity
+
+Let contexts take values in a measurable space \(\mathcal{C}\), and suppose the per-context parameter is \(\theta(c) \in \Theta\). For observation \((x,y,c)\), consider a conditional model \(p_\theta(y\mid x,c)\) with loss \(\ell(\theta; x,y,c)\). For a context neighborhood \(\mathcal{N}_\delta(c) = \{c': d(c,c') \le \delta\}\) under metric \(d\), define the effective sample size available to estimate \(\theta(c)\) by
+\[
+N_\text{eff}(c,\delta) \,=\, \sum_{i=1}^n w_\delta(c_i,c),\quad w_\delta(c_i,c) \propto K\!\left(\tfrac{d(c_i,c)}{\delta}\right),\ \sum_i w_\delta(c_i,c)=1,
+\]
+where \(K\) is a kernel. A kernel-regularized estimator with smoothness penalty \(\mathcal{R}(\theta)=\int \|\nabla_c \theta(c)\|^2\,\mathrm{d}c\) solves
+\[
+\widehat{\theta} \,=\, \arg\min_{\theta\in\Theta}\; \frac{1}{n}\sum_{i=1}^n \ell(\theta; x_i,y_i,c_i) \, + \, \lambda\, \mathcal{R}(\theta).
+\]
+Assuming local Lipschitzness in \(c\) and \(L\)-smooth, \(\mu\)-strongly convex risk in \(\theta\), a standard bias–variance decomposition yields for each component \(j\)
+\[
+\mathbb{E}\big[\|\widehat{\theta}_j(c)-\theta_j(c)\|^2\big] \;\lesssim\; \underbrace{\tfrac{\sigma^2}{N_\text{eff}(c,\delta)}}_{\text{variance}}\; +\; \underbrace{\delta^{2\alpha}}_{\text{approx. bias}}\; +\; \underbrace{\lambda^2}_{\text{reg. bias}},\quad \alpha>0,
+\]
+which exhibits the adaptivity–data trade-off: finer locality (small \(\delta\)) increases resolution but reduces \(N_\text{eff}\), inflating variance. Practical procedures pick \(\delta\) and \(\lambda\) to balance these terms (e.g., via validation), and amortized approaches replace \(\theta(c)\) by \(f_\phi(c)\) with shared parameters \(\phi\) to increase \(N_\text{eff}\) through parameter sharing.
+
+For computation, an early-stopped first-order method with step size \(\eta\) and \(T(c)\) context-dependent iterations satisfies (for smooth, strongly convex risk) the bound
+\[
+\mathcal{L}(\theta^{(T(c))}) - \mathcal{L}(\theta^*) \;\le\; (1-\eta\mu)^{T(c)}\,\big(\mathcal{L}(\theta^{(0)})-\mathcal{L}(\theta^*)\big) \, + \, \tfrac{\eta L\sigma^2}{2\mu N_\text{eff}(c,\delta)},
+\]
+linking compute allocation \(T(c)\) and data availability \(N_\text{eff}(c,\delta)\) to the attainable excess risk at context \(c\).

content/06.applications.md

@@ -0,0 +1,43 @@
+## Applications, Case Studies, and Evaluations
+
+### Implementation Across Sectors
+TODO: Detailed examination of context-adaptive models in sectors like healthcare and finance.
+
+### Performance Evaluation
+TODO: Successes, failures, and comparative analyses of context-adaptive models across applications.
+
+### Context-Aware Efficiency in Practice
+
+The principles of context-aware efficiency find practical applications across diverse domains, demonstrating how computational and statistical efficiency can be achieved through intelligent context utilization.
+
+In healthcare applications, context-aware efficiency enables adaptive imaging protocols that adjust scan parameters based on patient context such as age, symptoms, and medical history, reducing unnecessary radiation exposure. Personalized screening schedules optimize screening frequency based on individual risk factors and previous results, while resource allocation systems efficiently distribute limited healthcare resources based on patient acuity and context.
+
+Financial services leverage context-aware efficiency principles in risk assessment by adapting risk models based on market conditions, economic indicators, and individual borrower characteristics. Fraud detection systems use context-dependent thresholds and sampling strategies to balance detection accuracy with computational cost, while portfolio optimization dynamically adjusts rebalancing frequency based on market volatility and transaction costs [@ang2014asset].
+
+Industrial applications benefit from context-aware efficiency through predictive maintenance systems that adapt maintenance schedules based on equipment context including age, usage patterns, and environmental conditions [@lei2018machinery]. Quality control implements context-dependent sampling strategies that focus computational resources on high-risk production batches, and inventory management uses context-aware forecasting to optimize stock levels across different product categories and market conditions.
+
+A notable example of context-aware efficiency is adaptive clinical trial design, where trial parameters are dynamically adjusted based on accumulating evidence while maintaining statistical validity. Population enrichment refines patient selection criteria based on early trial results, and dose finding optimizes treatment dosages based on individual patient responses and safety profiles. These applications demonstrate how context-aware efficiency principles can lead to substantial improvements in both computational performance and real-world outcomes.
+
+### Formal metrics and evaluation
+
+Let \(\mathcal{C}\) denote the context space and \(\mathcal{D}_\text{test}\) a test distribution over \((x,y,c)\). For a predictor \(\hat{f}\), define the context-conditional risk
+\[
+\mathcal{R}(\hat{f}\mid c) \,=\, \mathbb{E}[\, \ell(\hat{f}(x,c), y) \mid c \,],\quad \mathcal{R}(\hat{f}) \,=\, \mathbb{E}_{c\sim \mathcal{D}_\text{test}}\, \mathcal{R}(\hat{f}\mid c).
+\]
+A context-stratified evaluation reports \(\mathcal{R}(\hat{f}\mid c)\) across predefined bins or via a smoothed estimate \(\int \mathcal{R}(\hat{f}\mid c)\,\mathrm{d}\Pi(c)\) for a measure \(\Pi\).
+
+Adaptation efficiency for a procedure that adapts from \(k\) in-context examples \(S_k(c)=\{(x_j,y_j,c)\}_{j=1}^k\) is
+\[
+\mathrm{AE}_k(c) \,=\, \mathcal{R}(\hat{f}_0\mid c) \, - \, \mathcal{R}(\hat{f}_{S_k}\mid c),\quad \mathrm{AE}_k \,=\, \mathbb{E}_{c}\, \mathrm{AE}_k(c),
+\]
+where \(\hat{f}_0\) is the non-adapted baseline and \(\hat{f}_{S_k}\) the adapted predictor. The data-efficiency curve \(k\mapsto \mathrm{AE}_k\) summarizes few-shot gains.
+
+Transfer across contexts \(\mathcal{C}_\text{src}\to \mathcal{C}_\text{tgt}\) with representation \(\phi\) can be measured by
+\[
+\mathrm{TP}(\phi) \,=\, \mathcal{R}_{\mathcal{C}_\text{tgt}}\big(\hat{f}_{\phi}\big) \, - \, \mathcal{R}_{\mathcal{C}_\text{tgt}}\big(\hat{f}_{\text{scratch}}\big),
+\]
+quantifying performance retained by transferring \(\phi\) versus training from scratch. Robustness to context shift \(Q\) is
+\[
+\mathrm{RS}(\hat{f};Q) \,=\, \sup_{\widetilde{\mathcal{D}}\in Q}\; \Big( \mathcal{R}_{\widetilde{\mathcal{D}}}(\hat{f}) - \mathcal{R}_{\mathcal{D}_\text{test}}(\hat{f}) \Big),
+\]
+where \(Q\) encodes permissible shifts (e.g., f-divergence or Wasserstein balls over context marginals).

content/10.conclusion.md

@@ -0,0 +1,37 @@
+## Conclusion
+
+### Overview of Insights
+TODO: Summarizing the main findings and contributions of this review.
+
+### Future Directions
+TODO: Discussing potential developments and innovations in context-adaptive statistical inference.
+
+### Context-Aware Efficiency: A Unifying Framework
+
+The principles of context-aware efficiency emerge as a unifying theme across the diverse methods surveyed in this review. This framework provides a systematic approach to designing methods that are both computationally tractable and statistically principled.
+
+Several fundamental insights emerge from our analysis. Rather than being a nuisance parameter, context provides information that can be leveraged to improve both statistical and computational efficiency. Methods that adapt their computational strategy based on context often achieve better performance than those that use fixed approaches. The design of context-aware methods requires careful consideration of how to balance computational efficiency with interpretability and regulatory compliance.
+
+Future research in context-aware efficiency should focus on developing methods that can efficiently handle high-dimensional, multimodal context information, creating systems that can adaptively allocate computational resources based on context complexity and urgency, investigating how efficiency principles learned in one domain can be transferred to others, and ensuring that context-aware efficiency methods can be deployed in regulated environments while maintaining interpretability.
+
+The development of context-aware efficiency principles has implications beyond statistical modeling. More efficient methods reduce computational costs and environmental impact, enabling sustainable computing practices. Efficient methods also democratize AI by enabling deployment of sophisticated models on resource-constrained devices. Furthermore, context-aware efficiency enables deployment of personalized models in time-critical applications, supporting real-time decision making.
+
+### Formal optimization view of context-aware efficiency
+
+Let \(f_\phi\!:\!\mathcal{X}\!\times\!\mathcal{C}\to\mathcal{Y}\) be a context-conditioned predictor with shared parameters \(\phi\). Given per-context compute budgets \(T(c)\) and a global regularizer \(\Omega(\phi)\), a resource-aware training objective is
+\[
+\min_{\phi}\; \mathbb{E}_{(x,y,c)\sim \mathcal{D}}\, \ell\big(f_\phi(x,c),y\big) \, + \, \lambda\,\Omega(\phi) \quad \text{s.t.}\quad \mathbb{E}_{c}\, \mathcal{C}\big(f_\phi; T(c), c\big) \le B,
+\]
+where \(\mathcal{C}(\cdot)\) models compute/latency. The Lagrangian relaxation
+\[
+\min_{\phi}\; \mathbb{E}_{(x,y,c)}\, \ell\big(f_\phi(x,c),y\big) + \lambda\,\Omega(\phi) + \gamma\, \mathbb{E}_{c}\, \mathcal{C}\big(f_\phi; T(c), c\big)
+\]
+trades off accuracy and compute via \(\gamma\). For mixture-of-experts or sparsity-inducing designs, let \(\phi=(\phi_1,\ldots,\phi_M)\) and a gating \(\pi_\phi(m\mid c)\). A compute-aware sparsity penalty is
+\[
+\Omega(\phi) \,=\, \sum_{m=1}^M \alpha_m\,\|\phi_m\|_2^2 \, + \, \tau\, \mathbb{E}_{c}\, \sum_{m=1}^M \pi_\phi(m\mid c),
+\]
+encouraging few active modules per context. Under smoothness and strong convexity, the optimality conditions yield KKT stationarity
+\[
+\nabla_\phi \Big( \mathbb{E}\,\ell + \lambda\,\Omega + \gamma\,\mathbb{E}_c\,\mathcal{C} \Big) \,=\, 0, \quad \gamma\,\Big( \mathbb{E}_c\,\mathcal{C} - B \Big)=0, \quad \gamma\ge 0.
+\]
+This perspective clarifies that context-aware efficiency arises from jointly selecting representation sharing, per-context compute allocation \(T(c)\), and sparsity in active submodules subject to resource budgets.

content/manual-references.json

@@ -1214,6 +1214,61 @@
     "issued": {"date-parts": [[2024]]},
     "volume": "abs/2410.16531",
     "URL": "https://api.semanticscholar.org/CorpusID:273507537"
+  },
+  {
+    "id": "Kaplan2020ScalingLF",
+    "type": "manuscript",
+    "title": "Scaling Laws for Neural Language Models",
+    "author": [
+      {"family": "Kaplan", "given": "Jared"},
+      {"family": "McCandlish", "given": "Sam"},
+      {"family": "Henighan", "given": "Tom"},
+      {"family": "Brown", "given": "Tom B."},
+      {"family": "Chess", "given": "Benjamin"},
+      {"family": "Child", "given": "Rewon"},
+      {"family": "Gray", "given": "Scott"},
+      {"family": "Radford", "given": "Alec"},
+      {"family": "Wu", "given": "Jeffrey"},
+      {"family": "Amodei", "given": "Dario"}
+    ],
+    "issued": {"date-parts": [[2020]]},
+    "archive": "arXiv",
+    "eprint": "2001.08361",
+    "URL": "https://arxiv.org/abs/2001.08361"
+  },
+  {
+    "id": "Hoffmann2022TrainingCO",
+    "type": "manuscript",
+    "title": "Training Compute-Optimal Large Language Models",
+    "author": [
+      {"family": "Hoffmann", "given": "Jordan"},
+      {"family": "Borgeaud", "given": "Sebastian"},
+      {"family": "Mensch", "given": "Arthur"},
+      {"family": "Buchatskaya", "given": "Elena"},
+      {"family": "Cai", "given": "Trevor"},
+      {"family": "Rutherford", "given": "Eliza"},
+      {"family": "de Las Casas", "given": "Diego"},
+      {"family": "Hendricks", "given": "Lisa Anne"},
+      {"family": "Welbl", "given": "Johannes"},
+      {"family": "Clark", "given": "Aidan"},
+      {"family": "Hennigan", "given": "Tom"},
+      {"family": "Noland", "given": "Eric"},
+      {"family": "Millican", "given": "Katie"},
+      {"family": "van den Driessche", "given": "George"},
+      {"family": "Damoc", "given": "Bogdan"},
+      {"family": "Guy", "given": "Aurelia"},
+      {"family": "Osindero", "given": "Simon"},
+      {"family": "Simonyan", "given": "Karen"},
+      {"family": "Elsen", "given": "Erich"},
+      {"family": "Rae", "given": "Jack W."},
+      {"family": "Vinyals", "given": "Oriol"},
+      {"family": "Sifre", "given": "Laurent"}
+    ],
+    "issued": {"date-parts": [[2022]]},
+    "archive": "arXiv",
+    "eprint": "2203.15556",
+    "URL": "https://arxiv.org/abs/2203.15556",
+    "note": "cs.CL"
   }
 ]