Skip to content

Prompts for HardNegative versions and full versions are different #3280

@Samoed

Description

@Samoed

Describe the bug

As found in #3266, the hard negatives versions of tasks are missing prompts. This is because they weren’t added in #1236. It’s not a big difference, but some instruction models could show lower performance because of it. I found this problem for:

  • ClimateFEVERHardNegatives
  • DBPediaHardNegatives
  • FEVERHardNegatives
  • HotpotQAHardNegatives
  • QuoraRetrievalHardNegatives
  • RiaNewsRetrievalHardNegatives

To reproduce

Run tasks

Additional information

No response

Are you interested to contribute a fix for this bug?

Yes

Metadata

Metadata

Assignees

No one assigned

    Labels

    reproquestion and issues related to reproducibility

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions