PROMISE: Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates — Scientific Code Review

Paper: Zachary Frangella, Pratik Rathore, Shipu Zhao, Madeleine Udell · published paper · arXiv 2309.02014
Code: udellgroup/PROMISE · commit a1cfef7ab940

This review ran executable checks that compare the released code against what the paper states. Issues come first, grouped by importance: major, then minor. Where the released materials are too incomplete to check a claim, the finding says so in its text; such gaps are reproducibility limitations, not demonstrated errors. Findings whose fix is a direct revision of the paper's TeX or the released code include a Prompt to fix panel with a ready-to-use LLM prompt.

Major Findings

R1 [major] — SciPy Compatibility Failure in Three Preconditioners

The paper specifies executable SSN, NySSN, SASSN-C, and SASSN-R preconditioners alongside the ridge and logistic model oracles.

The released objectives, gradients, and normalized Hessian diagonals agree with independent formulas, and NySSN's inverse agrees with an independently formed inverse to 8.881784197001252e-16. Under SciPy 1.12.0, however, SSN, SASSN-C (SASSN), and SASSN-R (LESSN) each stop with AttributeError: Module 'scipy' has no attribute 'identity' before producing a direction.

Readers can rely on the checked model formulas and NySSN calculation, but three of the four published preconditioner paths are not executable in the reviewed environment. Historical result files do not establish that these current code paths still run.

Suggested fix: Replace scipy.identity with numpy.eye or scipy.sparse.identity as appropriate and publish tested Python/SciPy pins. Rerun all four preconditioner smoke tests and any results intended to be reproduced with the updated environment.

Work in the released PROMISE repository for “PROMISE: Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates” (JMLR case jmlr_23_1187, frozen arXiv v2). Fix the SciPy 1.12.0 compatibility defect that causes SSN, SASSN-C (class SASSN), and SASSN-R (class LESSN) to raise “AttributeError: Module 'scipy' has no attribute 'identity'” before constructing a direction. The affected calls are in preconditioners/ssn.py and at line 43 of the two sparse-sketch modules defining SASSN and LESSN. Decide separately for each call whether numpy.eye or scipy.sparse.identity preserves the intended dense or sparse matrix representation, and replace scipy.identity accordingly without changing the published normalization or Woodbury calculations. Add tested Python and SciPy version pins. Then rerun smoke tests for SSN, NySSN, SASSN-C, and SASSN-R, compare every resulting inverse application with an independently formed inverse, and rerun any published results that are intended to be reproducible from the updated environment.

R2 [major] — Incomplete Results for Solved-Problem Verification

The paper defines success using scikit-learn's F(w*), absolute objective suboptimality 10^{-4}, and a budget of 200 full-data passes for the 51-problem performance study.

The complete 51-by-16 released result population is not present. Among 800 inspected records, no finite trajectory is available for mushrooms with SketchyKatyusha and NySSN, SASSN-C, or SASSN-R.

The released materials therefore do not allow the solved-problem predicate or the complete medium-scale performance population to be checked. This is a missing-results gap, not evidence that the paper's success criterion or conclusions are wrong.

Suggested fix: Publish the complete 51-by-16 result population, including reference F(w*) values and either finite trajectories or explicit failure records for every method–preconditioner combination.

R3 [major] — Missing End-to-End Reproduction Materials

The paper reports timed recommendations and performance results for 51 medium-scale problems, three showcase datasets, and one HIGGS streaming experiment under a specified CPU, RAM, and BLAS-threading setup.

The released repository contains historical result files, but it does not include the raw inputs, author-specified package versions, or the paper-time BLAS environment needed for an end-to-end rerun. The generated compatibility environment supports smoke testing but does not establish the original experimental environment.

The released materials therefore do not allow the medium-scale, showcase, or streaming results to be checked end to end. This is a reproduction gap, not a demonstrated error in the reported results.

Suggested fix: Publish raw-input acquisition instructions and checksums together with the exact Python, SciPy, and BLAS environment. Provide executable commands for rerunning every reported experiment.

Minor Findings

R4 [minor] — Missing Final Regularity Interval

The paper describes 50 SketchySAGA epochs and plots q_j = \gamma_{u,j}/\gamma_{l,j}, reporting that q_j is approximately 1 after 20 epochs on eight datasets.

Each released upper/lower archive contains only 49 ratios because the code saves an interval only when the next preconditioner update begins and does not save the final interval before returning. The plotting notebook consumes those arrays directly. At archive index 19, the ratios range from 1.000173754606391 to 1.0204381079843965; the eight Table 10 solve epochs still match 15, 86, 94, 20, 71, 49, 39, and 28 exactly.

The omission affects the regularity archives and Figure 9, not the optimizer trajectories or Table 10. Because the paper gives no numerical tolerance for “approximately one,” the missing interval does not by itself establish that the qualitative statement is wrong.

Suggested fix: Append the final upper/lower interval after the optimization loops and regenerate all eight regularity archives and Figure 9. State a numerical tolerance for the paper's “approximately one” comparison.

Work in the released PROMISE repository for “PROMISE: Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates” (JMLR case jmlr_23_1187, frozen arXiv v2). Repair the regularity archive produced by regularity_experiments.py. The driver requests 50 epochs, but opt_global_gammas appends the previous interval at line 106 and returns at line 131 without appending the final precond_gamma_glob_list, leaving 49 values of q_j=gamma_u,j/gamma_l,j for each of a9a, gisette, ijcnn1, mushrooms, phishing, rcv1, real-sim, and w8a. Decide whether the intended Figure 9 contract is one q_j interval per requested epoch. If it is, flush the final upper and lower interval after the optimization loop; if it is not, revise the paper and Figure 9 caption to state explicitly that 50 optimization epochs yield 49 completed plotted intervals. In either case, revise the paper to define a numerical tolerance for “approximately one”; the current ratios at archive index 19 range from 1.000173754606391 to 1.0204381079843965. Rerun all eight regularity experiments, regenerate every regularity NPZ archive and Figure 9, verify the intended archive length, and confirm that the Table 10 solve epochs remain exactly 15, 86, 94, 20, 71, 49, 39, and 28 in the listed dataset order.