JACKSONVILLE, Fla. — Clay County Sheriff Darryl Daniels was a few minutes late for his news conference on Wednesday but appeared poised, confident and ready to get his message out about an arrest in a home invasion case. 

"The win in this situation is that all the suspects have been apprehended and everybody has been put in jail to be held accountable for the crimes," said Daniels. 

However, when it came to him being held accountable for the shame that now overshadows his office, the sheriff dodged reporters.

Daniels walked out of his news conference, refusing to answer questions.

At the center of the public scandal is an extramarital affair with a former co-worker.

It became public when his former employer, the Jacksonville Sheriff's Office, released the findings of an internal affairs investigation into Cierra Smith.

Smith confirmed having sexual relations with Daniels when he was a jail director.

It has rocked the department. On Your Side was told by a source "his high-level staff has lost confidence in him."

"I think it is going to be very hard for him to be effective going down the road," said former Clay County Commissioner Ronnie Robinson. 

The former commissioner at one time supported Daniels. 

"He absolutely has a big ego and it is disappointing," said Robinson.

The long-time Clay County resident and business owner described Clay County politics as a "blood sport." He said people may forgive but they may not forget. 

"It is fair to say that he is in trouble politically," he said.

Sheriff Daniels has since made a public apology in a written statement, but Robinson said it did not go far enough.

"I think it is a bad scandal," he said.

The former county commissioner said he has received a number of calls from the community about the sheriff, the scandal and his future.

"I would be surprised if it doesn't work against him greatly," he said.

One person has already filed to run against Daniels and another is expected to come forward.

The Florida Department of Law Enforcement is reportedly reviewing the allegations, to see if Daniels crossed the line and violated his fiduciary duty to the residents of Clay County.