Welcome to ProgressTalk! The OpenEdge database doesn't have any native password complexity or history rules; I assume you are talking about users in the _User table.
That said, best practice these days is to keep your user credentials stored outside of the database and define an external authentication domain. Have a look at the documentation for the CLIENT-PRINCIPAL object if you want to move in that direction.
If you have an application table in your database where you store users and hashed passwords, and if you have the ability to change your application's code, then you can certainly implement password complexity and history in business logic. But you will have to think about compensating controls to prevent access to that table from outside your application, e.g. a Procedure Editor connected to the database.